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Abstract 

A  key  function  of  a  host-based  intrusion  detection  sys¬ 
tem  is  to  monitor  program  execution.  Models  constructed 
using  static  analysis  have  the  highly  desirable  feature  that 
they  do  not  produce  false  alarms;  however,  they  may  still 
miss  attacks.  Prior  work  has  shown  a  trade-off  between 
efficiency  and  precision.  In  particular,  the  more  accurate 
models  based  upon  pushdown  automata  (PDA)  are  very 
inefficient  to  operate  due  to  non-determinism  in  stack  ac¬ 
tivity.  In  this  paper,  we  present  techniques  for  determiniz- 
ing  PDA  models.  We  first  provide  a  formal  analysis  frame¬ 
work  of  PDA  models  and  introduce  the  concepts  of  deter¬ 
minism  and  stack-determinism.  We  then  present  the  VP- 
Static  model,  which  achieves  determinism  by  extracting  in¬ 
formation  about  stack  activity  of  the  program,  and  the  Dyck 
model,  which  achieves  stack-determinism  by  transforming 
the  program  and  inserting  code  to  expose  program  state. 
Our  results  show  that  in  run-time  monitoring,  our  models 
slow  execution  of  our  test  programs  by  1%  to  135%.  This 
shows  that  reasonable  efficiency  needs  not  be  sacrificed  for 
model  precision.  We  also  compare  the  two  models  and  dis¬ 
cover  that  deterministic  PDA  are  more  efficient,  although 
stack-deterministic  PDA  require  less  memory. 


1.  Introduction 

A  typical  host-based  intrusion  detection  system  (HIDS) 
monitors  execution  of  a  process  to  identify  potentially  mali¬ 
cious  behavior.  An  anomaly  detection  HIDS  identifies  vari¬ 
ations  from  a  preconstructed  model  of  normal  program  be¬ 
havior.  Such  a  system  interposes  a  monitor  between  a  pro¬ 


cess  and  the  operating  system.  All  events  (usually  system 
calls)  that  flow  from  the  program  to  the  operating  system 
are  validated  against  the  model.  Events  that  do  not  conform 
to  the  model  are  rejected  by  the  monitor.  Figure  1  shows  a 
typical  HIDS  architecture. 

There  are  several  techniques  to  construct  the  program 
model  used  in  an  anomaly  detection  HIDS.  Learning -based 
techniques  [4, 5, 8, 12, 14, 15, 22, 26]  construct  the  program 
model  by  training  on  a  set  of  execution  traces.  Sometimes 
a  specification  of  the  program  provided  by  a  domain  expert 
is  also  used  as  a  program  model  [11].  This  paper  focuses  on 
program  models  constructed  using  static  analysis  [6,23,24]. 
In  the  context  of  static  analysis,  there  is  a  trade-off  between 
efficiency  and  precision.  Non-deterministic  finite  automa¬ 
ton  (NFA)  models  are  efficient  to  operate,  but  introduce  im¬ 
possible  paths  because  they  do  not  model  the  call-return  se¬ 
mantics  of  the  program.  Pushdown  automaton  (PDA)  mod¬ 
els  eliminate  impossible  paths  by  incorporating  the  pro¬ 
gram’s  stack,  but  they  are  inefficient  to  operate.  The  inef¬ 
ficiency  in  the  PDA  models  occurs  because  the  stack  ac¬ 
tivity  of  the  program  is  hidden  from  the  model  and  results 
in  non-determinism.  Therefore,  the  state  space  of  the  PDA 
model  can  become  prohibitively  large  during  operation.  We 
call  this  the  curse  of  non-determinism.  This  paper  formally 
presents  several  techniques  to  handle  this  problem.  Specifi¬ 
cally,  we  make  the  following  contributions: 

•  Formal  framework.  Formal  models  in  intrusion  de¬ 
tection  research  have  received  scant  attention,  and 
we  address  this  shortcoming.  Investigating  for¬ 
malisms  drives  the  discovery  of  why  certain  pro¬ 
gram  models  do  or  do  not  exhibit  reasonable  per¬ 
formance.  Our  primary  purpose  is  to  formally  an¬ 
alyze  recently  proposed  models  rather  than  to  in- 
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Figure  1.  Architecture  of  a  host-based  intrusion  detection  system. 


troduce  all-new  models.  A  commonality  of  these 
models  is  the  exposure  of  process  execution  state  be¬ 
yond  a  simple  system  call  stream.  For  example, 
Sekar  et  al.  [19]  proposed  using  program  counter  in¬ 
formation.  Feng  et  al.  [2]  and  Giffin  el  al.  [7]  ex¬ 
posed  the  stack  activity  of  a  program.  We  show 
that  a  context-free  language  (CFL)  is  homomor¬ 
phic  to  a  deterministic  CFL.  The  proof  of  this  result 
is  similar  to  that  of  Chomsky  [1]  and  provides  in¬ 
tuition  about  techniques  that  expose  program  state. 
Non-determinism  in  stack  activity  is  the  major  fac¬ 
tor  contributing  to  the  time  and  space  complexity  of 
operating  PDA  models.  Motivated  by  this  observa¬ 
tion,  we  define  a  stack-deterministic  PDA  model  in 
which  the  stack  activity  is  deterministic.  Section  3  dis¬ 
cusses  these  formalisms. 

•  Model  determinizing  techniques.  Techniques  for  de- 
terminizing  the  PDA  models  essentially  incorporate 
additional  program  state  (such  as  the  program  counter 
and  stack  activity)  into  the  model.  We  describe  two 
techniques  incorporating  additional  state  of  the  pro¬ 
gram.  In  the  obsen’ational  technique,  the  monitor  ex¬ 
tracts  the  relevant  information  from  the  program.  For 
example,  the  monitor  can  extract  information  about  the 
stack  activity  of  the  program  by  walking  the  call  stack. 
Our  VPStatic  model,  a  statically-constructed  variant  of 
the  VtPath  model  [2],  implements  this  technique.  The 
rewriting  or  instrumentation  technique  transforms  the 
program  to  introduce  additional  code  that  exposes  pro¬ 
gram  state.  For  example,  additional  system  calls  intro¬ 
duced  before  a  call  to  function  f  indicate  to  the  model 
that  a  call  to  f  is  about  to  happen.  Our  recent  Dyck 
model  implements  this  approach  [7].  We  also  compare 
the  observational  and  rewriting  approaches  to  deter¬ 
minizing  the  PDA  model.  Sections  4  and  5  present  the 
two  models. 

•  Evaluation.  Our  results  show  that  the  formalisms  of 
deterministic  and  stack-deterministic  push-down  au¬ 
tomata  enable  construction  of  context-sensitive  pro¬ 


gram  models  suitable  for  online  security  monitoring. 
The  VPStatic  automaton  operation  slows  execution  of 
our  test  programs  by  0%  to  17%,  although  the  unop¬ 
timized  stack  walking  algorithm  adds  up  to  80%  ad¬ 
ditional  overhead.  The  Dyck  model  is  slightly  less  ef¬ 
ficient  due  to  state  non-determinism,  with  overheads 
of  8%  to  135%.  However,  the  Dyck  model  has  a  com¬ 
pact  representation,  requiring  only  38%  to  49%  more 
memory  for  program  instrumentation  and  the  state  ma¬ 
chine.  These  results  vindicate  context  sensitive  mod¬ 
els,  showing  that  reasonable  efficiency  needs  not  be 
sacrificed  for  model  precision.  Complete  results  are 
given  in  Section  6. 

2.  Related  Work 

Significant  intrusion  detection  systems  research  has  fo¬ 
cused  upon  static  and  dynamic  analysis  techniques  to  au¬ 
tomatically  generate  program  models.  Wagner  et  al.  pro¬ 
duced  models  via  static  analysis  of  C  source  code  [23,24]. 
They  described  the  precise  abstract  stack  model,  which  is 
a  non-deterministic  pushdown  automaton  (PDA).  Due  to 
the  stack  state  maintained  in  a  PDA,  this  model  captured 
the  precise  call  and  return  behavior  of  function  calls.  Un¬ 
fortunately,  runtime  automaton  operation  in  the  monitor 
was  prohibitively  high  for  some  programs,  reaching  sev¬ 
eral  tens  of  minutes  per  transaction.  We  observed  similar 
results  with  PDA  models  extracted  using  static  binary  anal¬ 
ysis  of  SPARC  executables  [6].  Both  papers  concluded  that 
imprecise,  context-insensitive  models  must  be  used  for  rea¬ 
sonable  performance. 

However,  only  context-sensitive  models,  such  as  the 
PDA,  can  detect  the  impossible  path  exploits  described  by 
Wagner  et  al.  [23,24].  Such  attacks  force  control  flow  to  en¬ 
ter  a  function  from  one  call  site  but  to  return  to  a  different 
call  site,  presumably  in  a  portion  of  the  program  code  suit¬ 
able  for  the  attack.  A  context-sensitive  model  detects  this  il¬ 
licit  control  flow  by  modeling  the  state  of  a  program’s  call 
stack. 


Our  experience  indicates  that  severe  non-determinism  in 
this  stack  state  is  the  major  contributing  factor  to  the  time 
and  space  complexity  of  PDA  operation.  The  Dyck  model 
proved  this:  by  eliminating  non-determinism  on  stack  tran¬ 
sitions,  a  context-sensitive  model  could  be  efficiently  oper¬ 
ated  [7],  This  paper  formalizes  the  Dyck  model  and  proves 
that  it  is  a  stack  deterministic  PDA.  We  further  improve  the 
model  by  eliminating  the  additional  system  calls  required 
by  the  previous  model  construction.  Our  VPStatic  model 
goes  further.  It  is  a  fully  deterministic  PDA  and  requires  no 
modifications  to  the  original,  analyzed  binary. 

Dynamic  analysis  extends  the  original  work  of  For¬ 
rest  et  al.  [3]  and  constructs  a  program  model  based  upon 
observed  system  call  traces  from  numerous  training  runs 
[4,5,8, 12, 14, 15,22,26,27].  Sekar  et  al.  showed  that  learning 
a  deterministic  automaton  is  possible  by  associating  each 
system  call  with  its  corresponding  program  counter  [19]. 
This  model  does  not  monitor  context  information  and  may 
miss  attacks  due  to  this  imprecision.  It  may  also  miss  at¬ 
tacks  on  dynamically  linked  libraries  due  to  its  oversimpli¬ 
fied  handling  of  dynamic  objects. 

Our  previous  VtPath  model  improved  the  precision  of 
dynamically  constructed  models  [2],  This  model  addition¬ 
ally  monitors  return  addresses  on  the  call  stack.  Our  VP- 
Static  model  is  a  natural  extension  of  VtPath  constructed 
using  static  analysis  techniques.  Again,  we  add  formalism 
to  the  previous  work.  The  VtPath  model  calculates  an  ad- 
hoc  virtual  path  from  the  call  stacks  of  two  adjacent  system 
calls  and  verifies  the  validity  of  that  path.  VPStatic  is  a  prov- 
ably  deterministic  PDA.  The  use  of  an  automaton  localizes 
transitions  to  states,  making  VPStatic  more  precise.  More¬ 
over,  the  VPStatic  model  does  not  suffer  the  false  alarms  of 
VtPaths  due  to  its  conservative  static  analysis. 

Others  have  pioneered  work  outside  of  static  and  dy¬ 
namic  analysis.  These  approaches  monitor  execution  based 
upon  specifications  of  system  calls  [10]  or  of  expected  pro¬ 
gram  behavior  [11],  When  provided  by  a  domain  expert, 
these  specifications  can  likely  enhance  the  quality  of  auto¬ 
matically  generated  models. 

The  VPStatic  model  has  an  anomaly  recovery  property 
not  considered  by  previous  approaches.  After  an  anomaly 
occurs,  we  can  still  uniquely  determine  the  expected  state 
and  stack  context  for  the  next  valid  system  call  by  monitor¬ 
ing  its  program  counter  and  call  stack.  Thus,  we  can  con¬ 
tinue  to  operate  the  automaton  and  potentially  detect  more 
attacks  such  as  a  root-level  exploit  following  a  probe.  This 
also  allows  for  a  greater  variety  of  security  policies  by  en¬ 
abling  the  system  to  fail  an  anomalous  system  call  and  con¬ 
tinue  execution  rather  than  terminate  the  program.  For  ex¬ 
ample,  a  monitor  that  terminates  a  network  daemon  after 
an  anomalous  system  call  could  be  used  for  a  denial  of  ser¬ 
vice  attack.  We  can  instead  prevent  just  the  anomalous  call 
and  allow  process  execution  and  monitoring  to  continue. 


3.  Formal  Models 

We  begin  by  formally  describing  pushdown  automata, 
deterministic  pushdown  automata,  and  stack-deterministic 
pushdown  automata.  These  finite  state  machines  are  the  un¬ 
derlying  constructs  of  our  program  models  used  for  intru¬ 
sion  detection. 

Definition  1  [PDA  and  DPDA ] 

A  pushdown  automaton  (PDA)  P  is  7-tuple 
P  =  ( Q ,  E,  r,  6,  qo ,  zq,  F),  where  Q  is  the  set  of  states,  E 
is  the  input  alphabet,  T  is  the  stack  alphabet,  5  is  the  tran¬ 
sition  relation  mapping  Q  X  (E  U  {e})  X  T  to  finite  sub¬ 
sets  of  Q  X  (r  U  {e})*,  <70  £  Q  is  the  unique  initial  state, 
zo  £  r  is  the  initial  stack  start  symbol,  and  F  C  Q  is 
the  set  of  accepting  states.  There  are  three  types  of  transi¬ 
tions  in  6: 

1.  (Input  consumption  or  e  transition):  (p,  z)  £  S  (q,  a ,  z) 
where  a  £  E  U  {e}. 

The  top  of  the  stack  is  z  and  stack  contents  do  not 
change.  If  a  =  e,  then  this  represents  a  transition  from 
q  to  p  that  consumes  no  input.  If  a  £  T,  and  P  is  in 
state  q,  then  consume  input  a  and  move  to  state  p. 

2.  [Push  transition;  pushes  z'  onto  the  stack):  (p,  zz')  £ 
5  ( q ,  a,  z)  where  a  £  E  U  {e}. 

The  explanation  is  the  same  as  (1),  but  now  z'  is 
pushed  onto  the  stack. 

3.  (Pop  transition;  pops  z  from  stack):  (p,  e)  £  5  (q,  a,  z) 
where  a  £  S  U  {e}. 

The  explanation  is  the  same  as  (1),  but  now  z  is  popped 
from  the  stack. 

A  PDA  P  =  (Q,  E,  r,  6 ,  qo,  Zq,  F )  is  called  determinis¬ 
tic  if  the  transition  relation  S  satisfies  the  following  condi¬ 
tions  [9]: 

•  (Condition  1):  For  all  q  £  Q  and  z  £  T,  whenever 
S(q ,  e,  z)  is  nonempty,  then  5{q,  a ,  z)  is  empty  for  all 
a  £  E. 

•  (Condition  2):  For  all  q  in  Q,  a  £  E  U  {e}  and  z  £V, 
S(q ,  a,  z)  contains  at  most  one  element. 

A  deterministic  PDA  is  abbreviated  as  DPDA. 

Our  definition  allows  only  one  stack  symbol  to  be  pushed 
onto  or  popped  from  the  stack.  The  most  general  definition 
of  a  PDA  (as  found  in  [9])  allows  more  than  one  stack  sym¬ 
bol  to  be  pushed  on  the  stack.  However,  it  is  easy  to  see 
that  a  PDA  P  (according  to  the  general  definition)  can  al¬ 
ways  be  converted  into  a  PDA  which  conforms  to  our  defini¬ 
tion  (the  construction  essentially  transforms  pushing  many 
symbols  on  the  stack  into  a  sequence  of  pushes  of  one  sym¬ 
bol.) 

Given  a  PDA  P  =  ( Q ,  E,  T,  S,  qo,  zo,  F ),  a  configura¬ 
tion  c  is  a  tuple  (■ q ,  7),  where  q  £  Q  is  the  current  state  and 


7  is  a  string  of  stack  symbols  representing  the  stack  con¬ 
tents.  Given  two  configurations  c  and  d  and  a  £  £,  we  say 
that  c  =>p  d  if  c  is  transformed  into  d  by  a  sequence  of 
transitions  of  the  PDA  P  while  consuming  input  a.  The  re¬ 
lation  =>p  can  be  extended  to  words  w  £  £*,  i.e.,  given  two 
configurations  c  and  d  and  w  £  £*,  c  =>p  d  if  c  is  trans¬ 
formed  into  d  by  a  sequence  of  transitions  of  the  PDA  P 
while  consuming  input  from  string  w.  When  P  is  clear  from 
the  context,  we  simply  write  instead  of  =>p.  The  lan¬ 
guage  L(P)  accepted  by  P  =  (Q,  £,  T,  <5,  q0,  zo,  F)  is  de¬ 
fined  as 

{u;|(go,zo)  =>p  (j>,  7)  for  some  p  £  F  and  7  £  T*} 

PDAs  accept  context  free  languages  (CFL).  If  a  language 
L  is  accepted  by  DPDA,  it  is  called  a  deterministic  con¬ 
text  free  language  or  DCFL.  Theorem  1  proves  that  every 
CFL  L  is  homomorphic  [9]  to  a  DCFL  V .  Moreover,  the 
proof  of  the  theorem  gives  a  procedure  for  determinizing  a 
PDA  by  expanding  the  input  alphabet.  This  proof  is  simi¬ 
lar  to  that  of  Chomsky  [1], 

Theorem  1  Let  L  be  a  CFL.  There  exists  a  DCFL  L p>  and 
a  homomorphism  h  such  that  h{Ljf)  =  L. 

Proof:  Let  P  =  (Q.  £,  1.  S.  qo,  zq.  F)  be  a  PDA  accepting 
L.  We  will  construct  a  new  input  alphabet  £p>.  There  are 
three  types  of  symbols  in  £  p>. 

•  Input:  For  each  a  £  £U{e}  and  p  £  Q,  there  is  an  in¬ 
put  symbol  ea,p.  The  input  symbol  ea>p  represents  con¬ 
suming  input  a  and  transitioning  to  state  p. 

•  Push:  For  each  a  €  £  U  {e},  p  £  Q  and  z  £  T,  there 
is  an  input  symbol  fa,p,z-  The  input  symbol  fa,p,z  rep¬ 
resents  consuming  input  a,  pushing  z  on  to  the  stack, 
and  transitioning  to  state  p. 

•  Pop:  For  each  aC£U{e},  p  £  Q  and  z  £  T,  there 
is  an  input  symbol  ga,p,z ■  The  input  symbol  ga,P,z  rep¬ 
resents  consuming  input  a,  popping  z  from  the  stack, 
and  transitioning  to  state  p. 

Next,  we  will  construct  a  DPDA  Pd  = 
(Q,  £p,  r,  6d,  <Zo;  zq,  F).  Notice  that  the  only  compo¬ 
nents  different  between  P  and  Pd  are  the  input  alpha¬ 
bet  and  the  transition  relation.  The  transition  relation  for 
the  DPDA  Pd  is  defined  as  follows: 

•  For  each  transition  ( p ,  z)  £  6  ( q ,  a,  z )  in  P,  we  have 
the  transition  6d  ( q ,  e0iP,  z)  =  {(p,  z)}. 

•  For  each  transition  ( p ,  zz')  £  6  ( q ,  a,  z)  in  P,  we  have 
the  transition  6d  ( q ,  fa,p,z z)  =  {(p,  zz')}. 

•  For  each  transition  ( p ,  e)  £  6  ( q ,  a,  z)  in  P,  we  have 
the  transition  6d  (q,  ga,p,zi  z)  =  {(p,  e)}. 


It  is  easy  to  see  that  Pd  is  deterministic.  Consider  the  fol¬ 
lowing  homomorphism  h: 


h(pa,p ) 

=  a 

(fa,p,z) 

—  a 

(ga,p,z) 

—  a 

Let  L{Pd)  be  the  language  accepted  by  the  DPDA  Pd. 
Then  h(L(PD))  =  L(P)  =  L.  □ 

The  construction  used  in  the  proof  of  Theorem  1  expands 
the  input  alphabet  by  exposing  the  stack  operations  and  the 
target  state  of  the  transition.  For  example,  the  input  sym¬ 
bol  f a,p,z  indicates  to  the  DPDA  Pd  that  it  should  consume 
input  a,  push  z  on  the  stack,  and  transition  to  state  p.  Sup¬ 
pose  that  a  PDA  P  models  a  program  Pr.  In  this  case,  the 
DPDA  Pd  models  the  program  Pr,  where  internal  state  of 
the  program  Pr  (such  as  stack  activity)  is  exposed.  In  other 
words,  exposing  program  state  corresponds  to  the  input  al¬ 
phabet  expansion  used  in  Theorem  1 . 

3.1.  Intrusion  Detection  using  PDAs  and  DPDAs 

In  model-based  intrusion  detection,  one  constructs  a 
model  M(Pr)  of  a  program  Pr  (see  Figure  1).  Pr  gen¬ 
erates  a  sequence  of  symbols  (usually  a  sequence  of  sys¬ 
tem  calls).  After  receiving  a  symbol  a  from  the  program, 
the  model  M(Pr )  determines  whether  there  exist  transi¬ 
tions  on  symbol  a.  If  there  does  not  exist  a  transition  on 
the  input  symbol  a,  the  monitor  reports  an  intrusion.  Other¬ 
wise,  the  model  processes  symbol  a  and  updates  its  state. 

PDA  models.  Suppose  that  the  model  M(Pr)  is  a  PDA 
(Q,  £,  r,  6,  qo,  zo,  F).  The  state  of  the  model  is  the  set  of 
configurations.  The  model’s  initial  state  is  {(<?oi  zo)}.  Let  C 
be  the  set  of  possible  configurations  for  M (Pr)  after  pro¬ 
cessing  a  sequence  of  symbols  w  from  Pr.  Suppose  the  next 
symbol  that  Pr  generates  is  a.  The  new  state  of  the  program 
is  succ(C,a ),  which  represents  all  configurations  that  re¬ 
sult  from  configurations  in  C  after  processing  input  a.  For¬ 
mally,  succ(C,a)  is  defined  as  {c'|3c  £  C.c  =>°  c'}.  If 
succ(C ,  a)  is  empty,  the  monitor  reports  an  intrusion.  Oth¬ 
erwise,  the  new  state  of  the  model  M (Pr)  is  succ(C,  a)  and 
the  processing  continues. 

In  general,  the  state  of  the  model  can  be  infinite.  For  ex¬ 
ample,  suppose  the  model  is  in  the  state  C  =  {(p,  2)}  and 
receives  a  symbol  a.  Assume  that  the  model  has  the  follow¬ 
ing  transitions: 

8(p,e,z)  =  {(p,zz)}  (1) 

S(p,a,z)  =  {(q,e)}  (2) 

It  is  easy  to  see  that  succ(C ,  a)  is  the  infinite  set  { (q,  zl)  i  > 
0}.  Notice  that  the  infiniteness  arises  from  rule  1,  which  cor¬ 
responds  to  left  recursion  in  a  program.  However,  it  turns 


out  that  the  state  of  the  model  (which  is  a  set  of  configura¬ 
tions)  is  regular  and  can  be  represented  as  a  finite-state  au¬ 
tomaton  [18,23].  The  time  and  space  complexity  of  updat¬ 
ing  the  state  of  the  model  after  receiving  a  symbol  is  unfor¬ 
tunately  cubic  in  the  size  of  the  model.  Wagner  and  Dean 
concluded  that  operating  a  PDA  model  for  intrusion  detec¬ 
tion  was  prohibitively  expensive  [23,24], 

DPDA  models.  Suppose  the  model  M(Pr)  is  a  DPDA. 
Given  an  input  symbol  a  G  E,  a  configuration  c,  and  a 
DPDA  M(Pr),  there  exists  at  most  one  configuration  d 
such  that  c  =>°  c! .  Therefore,  it  is  easy  to  see  that  during 
monitoring  the  set  of  configurations  C  has  at  most  one  con¬ 
figuration.  The  time  and  space  complexity  of  updating  the 
state  of  the  model  after  receiving  a  symbol  is  0(1). 

Stack-deterministic  PDA  models.  In  our  experience,  non¬ 
determinism  in  stack  activity  is  the  major  contributing  fac¬ 
tor  to  the  time  and  space  complexity  of  operating  PDA  mod¬ 
els.  This  motivates  our  definition  of  a  stack-deterministic 
PDA  model,  which  allows  non-determinism  but  requires 
the  state  of  the  stack  be  left  unchanged  at  points  of  non¬ 
determinism.  Formally,  a  PDA  P  =  (Q.  E,  I  .  S ,  q o,  zq •  F) 
is  called  a  stack-deterministic  PDA  or  sDPDA  if  it  satisfies 
the  following  two  conditions: 

•  (Condition  1):  No  stack  activity  on  e-transitions. 
There  is  no  push  or  pop  transition  S(q,  a,  z )  such  that 

a  =  e. 

•  (Condition  2):  Stack  activity  only  depends  upon  the 
input  symbol  and  the  top  of  the  stack. 

For  all  a  G  E  and  zeT,  there  does  not  exist  two  states 
qi  and  q2  (not  necessarily  different),  such  that 

(pi,wi)  G  S(qi,a,z), 

( P2,w2 )  G  S(q2,a,z), 

where  wi  fz  w2. 

Assume  that  we  use  an  sDPDA  model  M ( Pr )  of  a  pro¬ 
gram  Pr  for  intrusion  detection.  Let  C  be  the  set  of  config¬ 
urations  obtained  after  processing  a  sequence  of  symbols  w. 
From  the  two  conditions  given  above,  all  configurations  in 
C  must  have  the  same  stack.  Formally,  C  G  2^  x  I’*,  where 
Q  and  I  are  the  set  of  states  and  stack  alphabets  for  the 
model  M(Pr).  Since  the  size  of  C  can  be  at  most  n  =  \Q\, 
the  time  and  space  complexity  for  processing  a  new  sym¬ 
bol  a  is  0(n).  If  a  language  L  is  accepted  by  sDPDA,  it 
is  called  a  stack-deterministic  context  free  language  or  sD- 
CFL.  Theorem  3  in  Appendix  A  proves  that  the  language 
accepted  by  a  sDPDA  is  a  DCFL.  Therefore,  an  sDPDA  is 
not  fundamentally  more  powerful  than  a  DPDA. 

Theorem  2  Let  L  be  a  CFL.  There  exists  a  sDCFL  L \>  and 
a  homomorphism  h  such  that  h{Lif)  =  L. 


Proof:  Let  P  =  ( Q ,  E,  T.  5,  (jo,  Zq.  F )  be  a  PDA  accepting 
L.  We  will  construct  a  new  set  of  input  symbols  Es£>.  There 
are  three  types  of  symbols  hi  Es£>. 

•  Input:  This  is  simply  the  input  alphabet  E  of  the  PDA 
P. 

•  Push:  For  each  a  G  E  U  {e}  and  z  €  T,  there  is  an  in¬ 
put  symbol  /0j2.  The  input  symbol  /0jZ  represents  con¬ 
suming  input  a  and  pushing  z  onto  the  stack. 

•  Pop:  For  each  a  G  E  U  {e}  and  z  €  T,  there  is  an  in¬ 
put  symbol  ga)Z.  The  input  symbol  gChZ  represents  con¬ 
suming  input  a  and  popping  zfrom  the  stack. 

Next,  we  will  construct  a  sDPDA  Ps  d  = 
(Q,T,SD,F,6sD,qo,z;o,F).  Notice  that  the  only  com¬ 
ponents  different  between  P  and  Psd  are  the  input  alpha¬ 
bet  and  the  transition  relation.  The  transition  relation  for 
the  sDPDA  Psd  is  defined  as  follows: 

•  For  each  transition  ( p,z )  G  5(q,a,z)  in  P,  we  have 
the  transition  ( p ,  z)  G  Ssd  (<?,  a,  z ). 

•  For  each  transition  ( p ,  zz')  G  S  ( q ,  a ,  z)  in  P,  we  have 
the  transition  ( p ,  zz')  G  Ssd  (q,  fa.z z). 

•  For  each  transition  ( p ,  e)  G  S  ( q ,  a,  z)  in  P,  we  have 
the  transition  ( p ,  e)  G  Ssd  ( q ,  ga,z ,  z). 

It  is  easy  to  see  that  Psd  is  stack-deterministic.  Consider 
the  following  homomorphism  h: 

h(a)  =  a 
h(fa,z )  =  a 

Hda,z )  =  a 

Let  L(PsD)  be  the  language  accepted  by  Psd-  Then 
h(L(PsD))  =  L{P)  =  L.  '  □ 

The  construction  used  in  the  proof  of  Theorem  2  expands 
the  input  alphabet  by  exposing  the  stack  operations.  For  ex¬ 
ample,  the  input  alphabet  /0j2  indicates  to  the  sDPDA  Pr) 
that  it  should  consume  input  a  and  push  z  onto  the  stack. 
Recall  that  in  the  proof  of  Theorem  1  we  expanded  the  in¬ 
put  alphabet  to  expose  the  stack  activity  and  the  target  of  the 
transition,  e.g.,  fa,P,z  indicated  that  it  should  consume  input 
a,  push  2  on  the  stack,  and  transition  to  state  p.  In  construct¬ 
ing  an  sDPDA,  we  exposed  the  stack  activity  of  the  PDA  but 
not  the  target  of  the  transition. 

Table  1  summarizes  the  time  and  space  complexity  of 
processing  a  new  symbol  for  the  three  models.  The  size  of 
input  alphabet  for  the  three  models  is  also  shown.  From 
Theorems  1  and  2  it  is  clear  that  the  size  of  the  input  al¬ 
phabets  for  DPDA  and  sDPDA  models  is  larger  than  for  the 
corresponding  PDA. 
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Model 

PDA 

Time  complexity 
0(nm z) 

Space  complexity 
0(nm O 

Input  alphabet  size 
k 

DPDA 

orn 

0(1) 

Q(knr) 

sDPDA 

h  ofr ) 

0(n) 

Table  1.  Time  and  space  complexities  for  pro¬ 
cessing  an  input  symbol  with  various  mod¬ 
els.  The  number  of  states  and  transitions  in 
the  model  are  denoted  by  n  and  m  respec¬ 
tively.  The  size  of  the  input  and  stack  alpha¬ 
bets  in  the  PDA  are  denoted  by  k  and  r  re¬ 
spectively. 


3.2.  Connection  to  Existing  Techniques 

Several  authors  have  proposed  exposing  program 
state  to  improve  the  precision  of  the  models.  For  ex¬ 
ample,  Sekar  et  al.  [19]  propose  using  program  counter 
information.  This  is  equivalent  to  expanding  the  input  al¬ 
phabet  to  expose  the  target  of  the  transition.  Giffin  et  al.  [7] 
and  Feng  et  al.  [2]  expose  the  stack  activity  of  a  program. 
In  our  context,  this  is  equivalent  to  expanding  the  input  al¬ 
phabet  by  exposing  the  stack  activity  (this  is  very  similar 
to  the  homomorphism  demonstrated  in  the  proof  of  Theo¬ 
rem  2).  Therefore,  the  formal  framework  of  PDA,  DPDA, 
sDPDA,  and  homomorphisms  provides  a  systematic  way  of 
understanding  and  evaluating  techniques  that  expose  addi¬ 
tional  program  state. 

4.  The  VPStatic  Model:  Determinizing  via 
Stack  Exposure 

The  VPStatic  model  is  a  statically-constructed  variant  of 
the  context-sensitive  VtPath  model  [2].  Like  its  dynamic 
counterpart,  it  uses  stackwalks  during  execution  to  deter¬ 
mine  the  call  stack  state  of  the  monitored  process.  Com¬ 
bined  with  program  counter  monitoring,  this  produces  the 
extra  symbols  necessary  to  fully  determinize  the  model. 

4.1.  Model  Generation  by  Static  Analysis 

The  VPStatic  model  is  generated  by  statically  analyzing 
the  binary  executable  of  a  program.  We  first  introduce  no¬ 
tation.  There  is  a  function  entry  state  Entry(f)  and  exit 
state  Exit(f)  for  each  function  /  in  the  executable,  sys¬ 
tem  call  state  S  for  each  system  call  instruction,  and  call 
site  entry  state  C  (state  right  before  the  call)  and  exit  state 
C'  (state  right  after  the  return)  for  each  function  call  site. 
Addr(S),  Addr(C),  and  Addr{C')  denote  the  address  of 
the  corresponding  system  call  or  function  call  instruction 
( Addr(C )  =  Addr(C')).  Func(a)  is  the  function  contain¬ 
ing  the  instruction  at  address  a. 


char*  filename; 
pid_t[2]  pid; 

int  prepare (int  index)  {  Entry (prepare) 

char  buf [20] ; 

pid[index]  =  getpidO;  S_getpid 

strcpy(buf,  filename); 
return  open (buf,  0_RDWR) ;  S_open 
}  Exit (prepare) 

void  action  ()  {  Entry (action) 

uid_t  uid  =  getuidO;  S_getuid 

int  handle; 
if  (uid  !  =  0)  { 

handle  =  prepare(l);  Cl,  Cl' 

read (handle,  ...);  S_read 

}  else  { 

handle  =  prepare (0);  CO,  CO' 

write (handle,  .  .  . )  ;  S_write 

} 

close (handle) ;  S_close 

}  Exit (action) 

Figure  2.  A  simple  code  fragment  example. 

We  use  a  simple  program  fragment,  shown  in  Figure  2, 
as  a  running  example.  The  automaton  and  the  left  side  list 
of  transitions  in  Figure  3  describe  a  non-deterministic  PDA 
for  the  example  program  that  is  quite  similar  to  the  call- 
graph  model  [24],  As  for  the  callgraph  model,  system  call 
numbers  are  the  only  observed  inputs  to  simulate  the  au¬ 
tomaton.  We  use  “none”  (or  more  commonly  e)  as  a  place 
holder  when  transitions  are  not  associated  with  any  system 
call.  This  PDA  is  non-deterministic  since  we  have  not  ex¬ 
posed  stack  activities  and  targets  of  transitions.  To  make  the 
PDA  deterministic,  we  extract  address  information  from  the 
binary  to  expose  internal  state.  The  automaton  and  the  right 
side  list  of  transitions  in  Figure  3  describe  the  final  DPDA. 

The  input  symbols  of  the  DPDA  have  the  forms  in  the 
proof  of  Theorem  1,  with  slight  modifications.  Namely,  for 
system  call  and  call  site  states,  we  use  Addr(p )  instead 
of  p  to  expose  the  state,  since  the  address  information  is 
what  we  can  extract  from  program  counter  and  call  stack 
when  dynamically  monitoring  program  executions.  For  ex¬ 
ample,  g(a ,  Addr(p),  z )  or  g(a,p,  z)  means  the  automaton 
consumes  the  input  symbol,  pops  z  from  the  stack,  and  tran¬ 
sitions  to  state  p.  This  is  equivalent  to  the  transition  ga,p,z 
used  in  the  proof  of  Theorem  1.  Other  symbols  can  be  sim¬ 
ilarly  explained.  All  three  forms  of  input  symbols  e(. . .), 
/(. . .)  and  g(. . .)  appear  in  Figure  3.  The  formal  models 
section  proved  this  pushdown  automaton  is  deterministic. 
A  detailed  description  of  the  model  is  in  Appendix  B. 

4.2.  Online  Detection  by  Dynamic  Monitoring 

After  the  profile  is  generated,  we  can  simulate  the  au¬ 
tomaton  during  online  program  monitoring.  When  each  sys- 


el :  getuid 
e2:  none 
e3:  none 
e4:  read 
e5:  write 
e6:  close 
e7:  close 
e8:  none 
e9:  getpid 
elO: open 
ell: none 

el 2:  none  Push(Addr(C1)) 
el 3:  none  Push(Addr(CO)) 
el  4:  none  Pop(Addr(C1')) 
el 5:  none  Pop(Addr(CO')) 


el :  e(getuid,  Addr(S_getuid)) 

e2:  e(none,  Addr(CI)) 

e3:  efnone,  Addr(CO)) 

e4:  ejread,  Addr(S_read)) 

e5:  efwrite,  Addr(S_write)) 

e6:  e(close,  Addr(S_close)) 

e7:  enclose,  Addr(S_close)) 

e8:  e(none,  Exit(action)) 

e9:  ejgetpid,  Addr(S_getpid)) 

elO:  e(open,  Addr(S_open)) 

el  1 :  e(none,  Exit(prepare)) 

el 2:  f(none,  Entry(prepare),  Addr(CI)) 

el 3:  fjnone,  Entry(prepare),  Addr(CO)) 

el 4:  g(none,  Addr(CI'),  Addr(CI')) 

el 5:  g(none,  Addr(CO'),  Addr(CO')) 


Figure  3.  The  PDA  and  VPStatic  DPDA  generated  for  the  code  example. 


e(none,  Exit(Func(arn^. 1))) 
g(none ,  am,  am) 
e(none,  Exit(Func(am ))) 
g(none,  am_i ,  am_i) 


e(none,  Exit(Func(ai. (-1))) 

g(none,  ai,  ai)  (3) 

e(none,bi )  (4) 

/(none,  Entry  (Func(bi+ 1)),  bi) 
e(none ,  bi+i) 
f  (none,  Entry (Func(bi+2)) ,  fy+i) 

e(none,  bn) 

f  (none,  Entry (Func(bn+i)),bn)  (5) 

e(sB,bn+ 1)  (6) 


Figure  4.  VPStatic  input  symbol  sequence 
generated  for  system  call  SB. 


tem  call  is  made,  we  extract  all  the  call  site  addresses  for 
the  functions  that  have  not  returned  yet  into  a  virtual  stack 
list  (VSL),  ordered  from  the  outermost  function  to  the  in¬ 
nermost  function.  The  definition  of  VSL  is  similar  to  that 
in  [2]. 

Assume  A  =  a\,a,2,  ■  ■  ■  ,am  and  B  =  61,62,  ■  ■  ■  ,bn 
are  the  virtual  stack  lists  of  the  last  and  the  current  system 
calls,  respectively.  Also,  assume  ss  is  the  current  system 
call  and  bn+i  is  its  address  (the  current  program  counter), 
and  sa  and  am+ 1  are  the  last  system  call  and  its  address,  re¬ 
spectively.  Suppose  l  is  the  first  index  for  A  and  B  so  that 
the  corresponding  items  are  not  equal,  namely,  a,  -  b,  for 
i  =  1, 2, —  1,  and  a;  7^  6;.  For  the  current  system  call, 
we  generate  a  sequence  of  input  symbols  and  feed  them  to 
the  automaton  one  by  one.  The  input  symbol  sequence  gen¬ 
erated  is  shown  in  Figure  4. 

For  example,  assume  an  ordinary  user  (not  root)  exe¬ 


cutes  the  example  program  and  runs  to  the  getpid  line 
in  Figure  2.  The  virtual  stack  list  A  here  should  look 
like  “ prefix ,  Addr(C  1)”,  where  prefix  is  a  sequence  of  ad¬ 
dresses  corresponding  to  the  functions  that  lead  to  action. 
The  system  call  Sa  is  getpid.  If  the  program  executes 
to  open,  the  virtual  stack  list  B  here  is  the  same  as  A 
since  the  call  stack  does  not  change,  and  system  call  SB 
is  open.  So  from  Figure  4,  the  symbol  sequence  gener¬ 
ated  is  e(open,  Addr(Sopen)),  which  successfully  leads  the 
automaton  to  the  next  state  Sopen.  However,  if  an  attacker 
overflows  a  buffer  using  strcpy,  she  could  change  the  re¬ 
turn  address  of  prepare  to  Addr(C 0),  to  gain  unautho¬ 
rized  write  access  to  the  file  after  prepare  returns.  In  that 
case,  the  virtual  stack  list  B  changes  to  “ prefix ,  Addr(C 0)”. 
Since  Addr(C 0)  7^  Addr{C  1),  from  Figure  4,  the  symbol 
sequence  generated  will  be: 


e(none,  Exit  (prepare)) 
g(none,  Addr(Cl),  Addr(Cl)) 

e(none,  Addr(C0))  (7) 

/(none,  Enti'y (prepare),  Addr (CO)) 
e(open,  Addr(Sopen)) 


However,  since  state  SgetPid  does  not  have  a  transition  asso¬ 
ciated  with  e(none,  Exit(prepare)),  an  alarm  will  be  trig¬ 
gered  and  the  intrusion  is  detected. 

After  all  input  symbols  generated  for  a  system  call  are 
processed,  the  current  state  should  be  the  state  correspond¬ 
ing  to  the  system  call,  and  the  current  automaton  stack  con¬ 
text  is  just  the  VSL  of  this  system  call.  Namely,  the  current 
state  and  stack  context  can  be  uniquely  decided  for  a  valid 
system  call.  If  there  is  no  corresponding  transition  to  fol¬ 
low  for  an  input  symbol,  then  anomalous  execution  indica¬ 
tive  of  an  intrusion  attempt  has  occurred. 


action  prepare 


Figure  5.  Local  Dyck  models. 


5.  The  Dyck  Model:  Determinizing  via  Instru¬ 
mentation 

As  shown  in  Section  3,  adding  stack  determinism  to  a 
PDA  requires  additional  alphabet  symbols  to  make  stack¬ 
modifying  transitions  deterministic.  Statically  constructed 
program  models  use  the  PDA  stack  to  model  the  running 
process’s  call  stack.  Stack  operations  then  occur  at  func¬ 
tion  call  sites  and  returns.  The  Dyck  model  [7]  uses  bi¬ 
nary  rewriting  to  insert  code  before  and  after  each  func¬ 
tion  call  site  to  generate  the  extra  symbols  needed  for  stack- 
determinism. 

5.1.  Static  Model  Construction 

The  Dyck  static  analyzer  reads  a  binary  program  image 
and  produces  both  an  Dyck  model  and  an  instrumented  ver¬ 
sion  of  the  binary.  This  requires  four  steps: 

1.  For  each  function,  construct  a  control  flow  graph 
(CFG). 

2.  Convert  each  CFG  into  a  local  model,  a  non- 
deterministic  finite  automaton  that  accepts  all  se¬ 
quences  of  function  calls  and  kernel  traps  that 
the  function  could  generate  under  correct  execu¬ 
tion. 

3.  Classify  function  calls  and  insert  code  around  func¬ 
tion  call  sites  to  generate  symbols  necessary  for  stack- 
determinism.  This  instrumentation  adds  new  events 
into  the  call  stream,  so  we  update  local  models  to 
match. 

4.  Combine  the  collection  of  modified  local  models  into  a 
single  sDPDA  modeling  the  entire  rewritten  program. 

Recall  that  Figure  2  shows  code  for  two  example  functions, 
prepare  and  action.  Although  we  show  C  source  code 
for  readability,  we  analyze  SPARC  binary  code. 

We  convert  each  function’s  CFG  into  a  local  model.  This 
is  straightforward:  a  CFG  is  already  a  non-deterministic  fi¬ 
nite  state  machine  with  all  edges  unlabeled.  If  a  basic  block 
contains  a  user  call  or  kernel  trap  site,  we  label  all  outgoing 


void  action  ()  { 

uid_t  uid  =  getuid ( ) ; 
int  handle; 
if  (uid  ! =  0)  { 

precall (A) ; 

handle  =  prepare (1); 

postcall (A) ; 

read (handle,  ...); 

}  else  { 

precall (B) ; 

handle  =  prepare (0); 

postcall (B) ; 

write (handle,  . . . ) ; 

} 

close (handle) ; 

} 

Figure  6.  Example  Code  With  Dyck  Instru¬ 
mentation.  Inserted  code  appears  in  bold¬ 
face. 


edges  of  that  block  with  the  call  name.  We  label  all  other 
edges  f  and  convert  all  basic  blocks  into  automaton  states. 
The  e-reduced  and  minimized  local  automata  for  the  exam¬ 
ple  code  are  shown  in  Figure  5.  Appendix  C.l  gives  the  for¬ 
mal  definition  of  a  local  model. 

Next,  we  add  edges  to  the  local  models  around  func¬ 
tion  call  transitions  that  model  the  call  stack  changes  occur¬ 
ring  at  runtime.  An  edge  before  each  call  transition  pushes 
a  unique  identifier  onto  the  PDA  stack  kept  in  the  runtime 
monitor;  an  edge  after  the  call  pops  that  identifier  off.  Each 
call  site  has  a  unique  push  and  pop  symbol,  so  the  moni¬ 
tor  can  differentiate  between  different  call  sites  to  the  same 
function.  The  NFA  local  models  are  now  PDAs. 

To  add  stack-determinism  to  these  PDA  models,  we  must 
add  symbols  to  the  event  stream  that  distinguish  each  stack 
operation.  The  analyzer  rewrites  the  binary  image  of  the 
program  by  inserting  a  history  stack  into  the  program’s  data 
space  and  adding  code  immediately  before  and  after  each 
call  site.  The  history  stack  records  stack  changes  occurring 
since  the  last  kernel  trap.  Precall  code  before  call  site  A 
pushes  the  symbol  fe<A  onto  the  history  stack.  If  the  call 
generates  a  kernel  trap  before  returning,  then  the  monitor 
reads  all  collected  symbols  from  the  history  stack  to  iden¬ 
tify  the  execution  path  followed  in  the  program.  If  the  call 
returns  without  generating  a  kernel  trap,  then  the  postcall 
code  pops  fe  A  from  the  history  stack  and  discards  it.  Oth¬ 
erwise,  it  adds  the  symbol  gf.A  to  the  history  stack.  Figure  6 
shows  the  rewritten  code  for  action  with  instrumented 
call  sites  to  prepare. 

Adding  code  instrumentation  at  recursive  call  sites  has 
potentially  high  runtime  cost.  We  add  neither  stack  tran¬ 
sitions  to  the  local  models  nor  code  to  the  binary  image 
around  call  sites  that  may  recurse. 

Lastly,  we  compose  the  collection  of  modified  local  au¬ 
tomata  at  points  of  function  calls  to  form  the  global  model 


Figure  7.  Dyck  model. 


of  the  entire  program.  The  analyzer  replaces  each  function 
call  transition  with  e-edges  entering  and  returning  from  the 
model  of  the  target  function.  Figure  7  shows  the  completed 
Dyck  model  for  the  example  functions.  Note  the  similar¬ 
ity  to  the  VPStatic  model  described  earlier.  Here,  the  input 
symbol  fe>A  additionally  pushes  the  identifier  A  onto  the 
PDA  stack.  The  symbol  gf  a  is  an  input  symbol  that  pops 
A.  Appendix  C.  1  formally  defines  the  model  using  language 
theory.  The  Dyck  model  is  an  sDPDA  (see  Appendix  C.2). 

5.2.  Runtime  Monitoring 

The  user  executes  the  rewritten  binary  in  her  security- 
critical  environment,  with  the  runtime  monitor  tracing  its 
execution  at  system  calls.  The  monitor  enforces  the  model, 
guaranteeing  that  process  execution  does  not  deviate  from 
the  possible  sequences  of  system  call  streams. 

Dyck  model  operation  is  more  straightforward  than  VP- 
Static  operation.  Although  the  model  is  a  PDA,  the  mon¬ 
itor  keeps  only  one  PDA  stack  due  to  stack-determinism. 
When  the  traced  process  generates  a  kernel  trap,  the  moni¬ 
tor  reads  all  saved  symbols  from  the  process’s  history  stack. 
Each  symbol  is  an  input  to  the  automaton  that  modifies  the 
stack  state  and  corresponds  to  a  return  from  or  a  call  to  an¬ 
other  function.  These  symbols  are  equivalent  to  the  virtual 
path  symbols  calculated  from  stack  walks  in  the  VPStatic 
model.  The  monitor  then  processes  the  kernel  trap  symbol, 
permitting  execution  only  if  the  symbol  has  a  valid  transi¬ 
tion  in  the  model. 

6.  Performance  Measurements 

To  compare  the  performance  of  the  VPStatic  and  Dyck 
models,  we  measured  two  costs  of  execution  monitoring. 
First,  we  measured  the  increase  in  execution  time  when 
monitoring.  Second,  we  calculated  the  increased  memory 
use  due  to  the  program  models  and  Dyck  instrumentation. 

We  analyzed  performance  for  three  test  programs.  Our 
tools  currently  build  models  only  for  statically-linked  pro¬ 
grams.  As  a  result,  the  set  of  test  programs  is  not  repre¬ 


sentative  of  those  with  greatest  security  concern,  although 
they  do  contain  a  mix  of  computation-intensive  and  syscall- 
intensive  programs.  Table  2  lists  the  three  programs  and 
workloads  and  statistics  for  each,  htzipd  is  a  proprietary 
implementation  of  httpd  which  is  the  only  httpd  imple¬ 
mentation  we  successfully  compiled  statically  under  So¬ 
laris. 

Execution  time  overheads  are  calculated  by  subtracting 
a  base  execution  time  from  monitored  execution  time.  All 
times  are  averaged  over  several  runs.  Execution  times  do 
not  include  setup  time  in  the  monitor  during  which  the  pro¬ 
gram  model  is  read  from  disk.  The  current  implementation 
of  both  the  VPStatic  and  Dyck  monitors  execute  in  user 
space  and  detect  system  call  events  via  Solaris  process  trac¬ 
ing.  To  better  evaluate  the  cost  of  operating  each  model,  the 
base  execution  time  is  measured  with  process  tracing  en¬ 
abled.  At  each  system  call  stop,  the  monitor  does  nothing 
but  resume  the  execution  of  the  stopped  process.  The  differ¬ 
ence  between  base  time  and  monitored  time  then  captures 
exactly  the  overhead  of  model  operation. 

We  calculated  memory  usage  similarly.  The  value  of  in¬ 
terest  is  the  increase  in  state  required  for  each  process.  In 
particular,  the  static  code  of  the  monitoring  process  is  of 
little  meaning  as  it  may  be  shared  among  all  audited  pro¬ 
cesses.  For  the  VPStatic  model,  we  compute  the  per  process 
state  by  taking  the  difference  between  memory  use  with  full 
auditing  enabled  and  with  an  empty  profile  loaded  with  au¬ 
diting  disabled.  The  memory  used  by  the  Dyck  model  also 
includes  the  cost  of  binary  code  inserted  into  the  original 
application. 

This  section  does  not  include  measurements  with 
the  previously  used  average  branching  factor  met¬ 
ric  [23],  This  metric  is  poorly  suited  for  measurements 
of  context-sensitive  languages  as  stack  transitions  enter¬ 
ing  system  call  wrapper  functions  obscure  the  reachable 
system  calls.  Lacking  an  appropriate  metric,  we  rely  in¬ 
stead  upon  our  theoretical  discussions  in  the  previous 
sections  as  an  evaluation  of  the  strength  of  our  mod¬ 
els.  Strength  metrics  may  be  applied  in  the  future  if 
new  research  develops  reasonable  measurement  algo¬ 
rithms. 

6.1.  Execution  Time  Overhead  Results 

Table  3  contains  execution  time  overheads  for  the  VP¬ 
Static  and  Dyck  models.  Base  execution  times  are  presented 
twice  because  differences  in  monitor  implementations  re¬ 
sult  in  somewhat  different  base  times.  Due  to  the  high  cost 
of  the  stack  walk  operation  in  the  VPStatic  model,  we  sep¬ 
arate  the  model’s  runtime  into  two  components:  the  time 
to  operate  the  automaton  and  the  time  to  perform  the  stack 
walk.  The  Dyck  model  does  not  walk  the  call  stack,  so  no 
such  separation  is  presented. 


Program 

Workload 

Instructions 

Functions 

Call  Sites 

htzipd 

Service  500  client  requests,  transferring  151.7  MB  in  total. 

110,096 

1455 

6928 

gzip 

Compress  a  23.5  MB  tar  file. 

57,271 

884 

2844 

cat 

Concatenate  38  files  totalling  500  MB  to  a  file. 

52,601 

838 

2728 

Table  2.  Test  programs,  workloads,  and  statistics. 
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Automaton 
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MB 
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htzipd 

16.66 
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3.80 

wm 

17.54 
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20.50 

27.72 

gzip 

13.72 

14.69 

0.03 

0.17 

i 

13.99 

1.17 

8 

cat 

46.20 

59.14 

2.56 

4 

15.81 

mm\ 

54.84 

30.60 

■aa 

Table  3.  Model  execution  times  in  seconds.  Base  execution  time  includes  system  call  tracing  without 
automaton  operation.  Percentages  compare  against  base  execution. 


Program 

VPStatic 

Dyck 

htzipd 

225 

300 

gzip 

389 

415 

cat 

170 

289 

Table  4.  Average  system  call  verification 
time,  in  microseconds. 


Table  4  shows  the  average  monitor  execution  time,  in  mi¬ 
croseconds,  per  system  call  event  received.  Each  system  call 
requires  the  monitor  to  update  its  calling  context  informa¬ 
tion  and  to  verify  that  the  system  call  is  a  valid  operation 
in  the  program  model.  The  times  to  perform  these  opera¬ 
tions  remained  relatively  constant  even  as  the  number  of 
stack  symbols  read  from  the  monitored  process  changed,  al¬ 
though  outlying  points  did  occur. 

The  tables  show  two  interesting  results.  First,  these  de¬ 
terministic  or  stack-deterministic  models  are  efficient  to  op¬ 
erate.  Automaton  operations  in  the  deterministic  VPStatic 
model  are  extremely  fast.  Second,  the  VPStatic  model  is 
more  efficient  to  operate  than  the  Dyck  model. 

This  second  result  occurs  for  two  reasons.  First,  it  illus¬ 
trates  the  operational  differences  between  deterministic  and 
stack-deterministic  automata.  The  DPDA  used  with  the  VP¬ 
Static  model  operates  in  constant  time,  but  the  sDPDA  un¬ 
derlying  the  Dyck  model  requires  linear  time  operation  (Ta¬ 
ble  1).  This  effect  is  clearly  visible  in  the  respective  run¬ 
times  of  the  two  models.  Second,  the  Dyck  model  has  addi¬ 
tional  execution  at  many  function  call  sites  due  to  the  in¬ 
jected  code.  This  cost  arises  even  if  the  process  follows 
an  execution  path  that  does  not  generate  system  calls.  The 
VPStatic  model  incurs  monitoring  cost  only  at  system  call 
events. 


6.2.  Memory  Use  Overhead  Results 

Table  5  presents  the  memory  needs  of  execution  moni¬ 
toring  for  the  two  models.  We  divide  the  memory  costs  of 
the  Dyck  model  into  the  cost  of  the  current  rewriting  in¬ 
frastructure,  which  doubles  the  size  of  each  program’s  code 
segment,  and  the  cost  of  our  code  insertions  and  state  ma¬ 
chine  representation.  The  infrastructure  cost  is  excessive, 
but  could  be  significantly  reduced  by  shifting  to  a  more  ef¬ 
ficient  rewriter. 

The  VPStatic  state  machine  cost  is  greater  than  the  cor¬ 
responding  Dyck  models.  Again,  this  highlights  differences 
between  DPDA  and  sDPDA  models.  An  automaton  al¬ 
lowing  non-determinism  in  state  transitions  naturally  has 
a  more  compact  representation.  Hence,  the  Dyck  model 
will  produce  smaller  automaton  structures  than  the  VPStatic 
model.  Moreover,  we  have  not  yet  optimized  the  VPStatic 
model  size.  For  example,  we  could  remove  all  the  function 
entry  and  exit  nodes  using  techniques  similar  to  the  automa¬ 
ton  reduction  used  for  Dyck  model.  We  kept  the  original  for¬ 
mat  of  the  model  since  it  is  a  recent  development  and  is  con¬ 
ceptually  clearer  this  way. 

6.3.  Discussion 

We  draw  two  primary  conclusions  from  this  work.  First, 
the  formalisms  of  deterministic  and  stack-deterministic 
push-down  automata  result  in  highly  accurate  and  highly 
efficient  program  models.  Non-deterministic  context- 
sensitive  models  produced  overheads  orders  of  magni¬ 
tude  worse  than  base  execution  [6,23,24]  and  would  never 
be  suitable  for  real-world  operation.  Our  automaton  oper¬ 
ation  overheads,  while  not  yet  as  low  as  we  would  like, 
show  that  context-sensitivity  and  precise  program  mod¬ 
els  need  not  be  sacrificed  for  performance. 

Second,  the  differences  in  these  models  suggests  that  hy¬ 
bridization  of  the  two  construction  and  monitoring  tech- 
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mm 
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Table  5.  Memory  use  in  KB  due  to  monitoring.  Percentages  are  increases  over  unmonitored  execu¬ 
tion. 


niques  may  be  beneficial.  The  Dyck  model  produces  no 
context  information  at  points  of  recursion  or  dynamic  link¬ 
ing  to  non-instrumented  binaries.  The  VPStatic  model  can 
identify  this  missing  information  by  inspecting  existing  pro¬ 
gram  state.  If  instrumented  libraries  are  available,  the  Dyck 
model  can  more  easily  use  these  libraries  at  runtime  as 
memory  offsets  of  return  addresses  are  not  an  issue.  The 
Dyck  model  can  also  successfully  reveal  context  informa¬ 
tion  in  optimized  binaries  where  stack  walking  may  be  dif¬ 
ficult  or  impossible.  On  the  other  hand,  binary  rewriting 
does  occasionally  fail.  We  can  then  rely  on  the  stack  walk 
technique  to  recover  state  information.  Likewise,  we  may 
wish  to  limit  instrumentation  to  some  set  of  critical  pro¬ 
gram  points  and  rely  upon  stack  walking  elsewhere.  A  hy¬ 
brid  model  would  combine  both  state  recovery  mechanisms 
to  capture  the  complete  context  of  a  system  call.  The  hybrid 
would  gain  from  the  strengths  of  both  models  while  mini¬ 
mizing  the  drawbacks  of  each. 


7.  Limitations 

Although  our  approaches  produce  more  sensitive  and 
more  accurate  models  than  other  approaches,  there  are  still 
limitations.  It  is  well  documented  [16, 17,25]  that  attack¬ 
ers  can  exploit  weaknesses  and  limitations  of  intrusion  de¬ 
tection  models  to  avoid  detection.  Short  of  complete  in¬ 
strumentation,  which  amounts  to  essentially  interpreting 
the  program,  our  statically-generated  models  do  not  have 
complete  information  about  the  state  of  the  executing  pro¬ 
gram.  An  attacker  can  exploit  incomplete  information  in  the 
model  to  evade  the  HIDS. 


7.1.  Incomplete  Sensitivity 

Models  discussed  in  this  paper  incorporate  information 
about  the  stack  activity  of  the  program.  Thus,  our  models 
are  context  sensitive.  However,  since  our  model  does  not 
track  predicates  used  in  branches,  they  are  neither  flow  nor 
path  sensitive.  This  incompleteness  can  result  in  our  model 
allowing  extraneous  behavior.  For  example,  consider  the 
following  code  fragment: 


char  *str,  *user; 

str  =  (char  *)  calloc  (...); 
user  =  (char  *)  calloc  (...); 

if  (strncmp  (user,  "admin",  5))  { 

sys_l  (); 

}  else  { 

sys_2  ( ) ; 

} 

strcpy  (str,  someinput) ; 
if  (strncmp  (user,  "admin",  5))  { 

sys_3  ( ) ; 

}  else  { 

sys_4  ( ) ; 

} 

There  are  two  possible  system  call  sequences  for  the  code 
fragment: 

sys_l,  sys_3,  and 
sys_2,  sys_4. 

The  sequences  correspond  to  the  predicate 
strncmp  (user,  "admin",  5)  being  true  or  false  re¬ 
spectively.  Notice  that  the  predicates  used  in  both  the 
branches  are  the  same.  However,  since  our  models  do 
not  track  the  values  of  branch  predicates,  they  will  al¬ 
low  the  following  four  sequences: 

sys.l,  sys_3, 
sys.l,  sys_4, 
sys_2,  sys_3,  and 
sys_2,  sys_4. 

An  attacker  can  exploit  this  limitation  to  avoid  detec¬ 
tion.  In  the  example  given  above,  an  attack  uses  a  large 
someinput  in  strcpy  to  overflow  str  on  the  heap  to 
change  the  value  of  user.  If  user  is  “guest”  and  the 
overflow  in  strcpy  changes  user  to  “admin”,  then  the 
illegal  sequence  sys_l,  sys_4  is  executed,  which  is  ac¬ 
cepted  by  our  model  and  hence  the  attack  is  not  de¬ 
tected. 


7.2.  Incomplete  Set  of  Events 

Events  monitored  by  our  model  are  system  calls.  As 
pointed  out  by  Wagner  and  Soto  [25]  and  Tan  et  al.  [20,21], 
an  attacker  can  evade  detection  if  it  generates  sequence 
of  events  accepted  by  our  model.  We  previously  presented 
such  an  attack  using  the  following  code  segment  [2]: 


f();  //f  has  no  system  calls,  a  buffer-overflow  occurs 
//so  that  after  f  the  program  jumps  to  IP 

if  (regular_user)  { 
return  ( ) ; 

} 

IP:  //super  user  privileges 
execve  ( ' '/bin/sh' ' ) ; 

An  attack  that  uses  a  buffer  overflow  in  f  to  force  the 
program  to  jump  to  IP  can  (illegally)  obtain  a  root  shell. 
Without  inserting  code  instrumentation  before  and  after  f , 
our  models  will  miss  this  attack  because  there  is  no  sys¬ 
tem  call  in  the  code  segment  between  the  call  to  f  and  IP. 
In  other  words,  no  matter  how  the  program  control  flow  is 
illegally  modified  within  the  code  segment,  there  is  no  ob¬ 
servable  events  to  our  models.  However,  when  code  instru¬ 
mentation  is  added  right  before  and  after  the  call  to  f ,  the 
attack  will  be  detected  by  our  model. 

An  attack  can  also  evade  detection  if  it  simply  replaces 
the  system  call  parameters  [25],  We  recover  some  argu¬ 
ments  of  system  calls  using  static  analysis,  but  there  are 
several  system  calls  where  we  have  incomplete  information 
about  the  arguments. 

7.3.  Playing  Inside  the  Sandbox:  Mimicry  Attacks 

We  assume  that  attackers  have  complete  knowledge 
about  our  model-construction  algorithm.  In  mimicry  at¬ 
tacks,  an  adversary  transforms  her  attack  in  such  a  way 
that  the  resulting  sequence  is  accepted  by  the  detec¬ 
tion  model  [25].  For  example,  the  attacker  can  mimic 
the  legal  program  behavior  by  generating  (legal)  sys¬ 
tem  calls  and  inserting  them  in  the  original  attack  se¬ 
quence.  An  attacker  can  use  a  different  attack  sequence 
that  is  semantically  equivalent  to  the  original  attack  se¬ 
quence.  These  attacks  are  very  serious  and  can  easily  evade 
simple  detection  models,  such  as  the  n-gram  model  pro¬ 
posed  by  [3],  However,  incorporating  additional  informa¬ 
tion  about  the  program  in  the  model  makes  it  difficult  to 
mount  mimicry  attacks.  Our  models  monitors  informa¬ 
tion  about  system  calls,  program  counter,  and  call  stack. 
Therefore,  to  mount  a  successful  mimicry  attack  an  adver¬ 
sary  is  required  to  produce  correct  call  stack  and  program 
counter  information  along  with  the  sequence  of  sys¬ 
tem  calls  which  is  equivalent  to  the  attack. 

8.  Conclusions 

We  formally  described  statically-constructed  context- 
sensitive  program  models  for  host-based  intrusion  detec¬ 
tion.  Seeking  to  add  efficiency  to  the  precision  of  these 
models,  we  examined  deterministic  PDAs  and  introduced 
stack-deterministic  PDAs.  The  proofs  of  language  equiva¬ 
lence  between  the  homomorphic  image  of  a  DCFL  or  sD- 
CFL  and  a  CFL  give  rise  to  monitoring  techniques  that 


make  these  models  possible.  The  VPStatic  model  walks  a 
process’s  call  stack  to  harvest  return  addresses  revealing 
context  information  enabling  a  deterministic  model.  Using 
program  instrumentation,  the  Dyck  model  eliminates  stack 
non-determinism.  Experiments  demonstrate  that  context- 
sensitivity  and  efficiency  can  coexist  in  these  program  mod¬ 
els,  benefiting  all  such  intrusion  detection  systems. 
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A.  Proof  for  Section  3 

Theorem  3  The  language  L  accepted  by  an  sDPDA  is  a 
DCFL. 

Proof:  First,  remove  e  transitions.  Let  P  = 

(Q,  E,  T,  5,  qo,  zo,  F)  be  an  sDPDA.  Recall  that  an  sD¬ 
PDA  does  not  have  stack  activity  on  e-transitions.  Given 
states  q  and  q'  and  a  stack  symbol  z  G  T,  we  say  that 
(q,  z)  (q1 ,  z)  if  there  exists  an  e-transition  such  that 
( q' ,  z)  €  S(q,  e,  z).  Let  =#*  be  the  reflexive  and  transi¬ 
tive  closure  of  =#e.  Notice  that  =>*  can  be  computed  in 
polynomial  time  using  standard  graph  reachability  al¬ 
gorithms.  We  will  transform  the  transition  relation  5 
of  P  to  S1  to  remove  e  transitions.  First,  S1  will  con¬ 
tain  all  the  non-e  transitions  S(q,a,z).  For  each  transi¬ 
tion,  ( q',z )  G  S(q,a,z)  (where  a  e),  S'(q,a,z )  contains 
all  (q" ,  z)  such  that  (q' ,  z)  =#*  (q" ,  z). 

Second,  remove  state  non-determinism.  Due  to  the 
step  given  above,  we  can  assume  that  the  sDPDA 
P  does  not  contain  e-transitions.  Next  we  can  re¬ 
move  the  “state”  non-determinism  of  P  using  the 
standard  subset  construction  used  in  determinizing 
a  NFA.  Let  P  =  (Q,D,T,5,qo,zo,F)  be  an  sD¬ 

PDA  without  e-transitions.  We  will  construct  a  DPDA 
DP  =  (Qi,  E,  T,  5\,qQ,Zo,  Fi),  where  Qi  =  2Q, 

9o  =  {do },  F\  are  all  subsets  of  Q  such  that  F\  n  F  ^  0, 
and  <5i  is  defined  as  follows:  5i(qi,a,z)  contains  (q2,z'), 
where  <72  is  the  following  set: 

W  I  €  <71  :  (</,  z’)  G  S(q,  a,  z )} 

Recall  that  because  of  condition  2  in  the  definition  of  sD¬ 
PDA  the  stack  activity  is  completely  determined  by  the  in¬ 
put  a  and  top  of  the  stack  z.  Therefore,  the  definition  of  5\ 


is  well  defined.  It  is  easy  to  see  that  DP  is  a  DPDA  and  ac¬ 
cepts  the  same  language  as  the  sDPDA  P.  □ 

B.  Definition  of  VPStatic  Model 

We  expand  the  notation  in  Section  4.  SysCall(S)  is  the 
system  call  made  at  S,  and  Target(C )  =  Target(C  )  is 
the  target  function  of  the  call  site  C /C' . 

The  computation  model  is  a  DPDA  (Q,  E,  T,  5,  qo,  zq,  F). 
We  still  use  the  simple  program  fragment,  shown  in  Fig¬ 
ure  2,  as  a  running  example.  Its  corresponding  automaton 
is  shown  in  Figure  3. 

Q  is  the  set  of  states.  The  example  program  has  14 
states.  Note  we  have  five  different  kinds  of  states  in  the  au¬ 
tomaton:  function  entry  states  ( Entry(action )  and 
Entry  (prepare)),  function  exit  states  ( Exit(action )  and 
Exit  (prepare)),  system  call  states  ( S-getuid ,  Spread  and 
so  on),  call  site  entry  states  (Cl  and  CO)  and  call  site  exit 
states  (Cl  and  CO  ). 

E  is  the  input  alphabet.  The  input  symbols  of  this  DPDA 
have  the  forms  in  the  proof  of  Theorem  1,  with  slight  mod¬ 
ifications.  Namely,  for  system  call  and  call  site  states,  we 
use  Addr(p)  instead  of  p  to  expose  the  state.  For  example, 
g(a,  Addr(p),  z)  or  g(a,p,z)  means  the  automaton  con¬ 
sumes  the  input  symbol,  pops  z  from  the  stack,  and  tran¬ 
sitions  to  state  p.  This  is  equivalent  to  the  transition  ga,P,z 
used  in  the  proof  of  Theorem  1 .  Other  symbols  can  be  simi¬ 
larly  explained  using  the  proof  of  Theorem  1 .  During  online 
detection,  we  monitor  the  call  stack  and  program  counter  to 
expose  address  information  Addr(p).  We  use  none  as  the 
placeholder  when  no  system  call  is  involved  for  the  tran¬ 
sition.  All  three  forms  of  input  symbols  e(. . .),  /(. . .)  and 
g(. . .)  appear  in  Figure  3. 

r  is  the  stack  alphabet.  For  our  model,  T  is 

{Addr(p)\p  £  Q  and p  is  a  function  call  site  state}  U  {zo} 

where  Zo  is  the  initial  stack  start  symbol.  We  use  the  au¬ 
tomaton  stack  of  the  DPDA  to  simulate  the  program  call 
stack. 

The  start  state  qo  is  the  entry  state  of  the  program  entry 
function.  F  is  the  set  of  accepting  states.  If  we  require  the 
program  to  exit  normally,  F  is  the  set  of  all  states  for  exit 
system  calls.  If  the  program  can  be  killed  anytime,  F  is  the 
set  of  all  states,  or  F  =  Q.  Since  the  example  program  is 
only  a  program  fragment,  the  start  and  the  accept  state  set 
are  not  shown  in  Figure  3. 

6  is  the  transition  function.  The  automaton  is  constructed 
by  interconnecting  the  transformed  control  flow  graph  of 
each  function.  If  both  the  states  connected  by  a  transition 
e  are  in  the  same  function,  we  call  e  an  intra-function 
transition.  Otherwise,  we  call  e  an  inter-function  transi¬ 
tion.  Intra-function  transitions  are  always  marked  with  in¬ 
put  symbols  of  the  form  e(...)  since  they  do  not  deal  with 


automaton  stack.  For  example,  in  Figure  3,  transition  e6: 
e(close,  Addr(S -dose))  means  that  if  the  current  state  is 
S-read,  the  program  issues  a  system  call  close,  and  we  ob¬ 
serve  that  its  program  counter  is  Addr(S -close),  then  the 
current  state  is  moved  to  S -close. 

Inter-function  transitions  modify  the  automaton  stack. 
They  only  exist  between  a  call  site  entry  state  and  its  target 
function  entry  state,  and  between  a  target  function  exit  state 
and  a  corresponding  call  site  exit  state.  If  Tj  is  a  call  site 
entry  state  and  T2  is  the  corresponding  target  function  en¬ 
try  state,  we  add  a  transition  from  Tj  to  T2  and  label  it  with 
f(none,T2,Addr(Ti)),  which  means  the  program  is  call¬ 
ing  a  function,  and  we  push  the  address  of  the  correspond¬ 
ing  call  site  into  the  automaton  stack.  In  Figure  3,  transi¬ 
tions  ei2  and  ei3  belong  to  this  case.  If  I)  is  a  call  site 
exit  state,  and  T\  is  the  corresponding  target  function  exit 
state,  we  add  a  transition  from  T\  to  T2  and  label  it  with 
g(none,  Addr(T2) ,  Addr(Tf)),  which  means  we  only  fol¬ 
low  this  transition  if  the  address  of  the  call  site  the  program 
is  returning  to  matches  the  top  symbol  on  automaton  stack, 
and  we  pop  this  address.  In  Figure  3,  transitions  ei4  and  eis 
belong  to  this  case. 

This  completes  our  model  definition.  The  formal  mod¬ 
els  section  proved  this  pushdown  automaton  is  determinis¬ 
tic.  Note  recursive  function  call  and  return  transitions  are 
handled  just  like  non-recursive  ones. 

C.  Definitions  and  Proofs  for  Section  5 

C.l.  Definition  of  Dyck  Model 

Let  S  be  the  set  of  system  call  sites  (traps  to  the  operat¬ 
ing  system)  and  C  be  the  set  of  function  call  sites.  Let  9(c) 
denote  the  target  function  of  call  site  c.  Note  that  two  differ¬ 
ent  call  sites  ci,  C2  £  C  are  unique,  even  if  6(c\)  =  6(c 2). 

Definition  2  [Local  Model] 

Let  G  =  (V,  E),  E  C  V  x  V  be  the  control  flow  graph 
of  program  Pr.  Let  a  <  v  indicate  that  vertex  v  £  V  con¬ 
tains  call  site  a.  The  local  model  for  each  function  i  £  Pr 
is  Ai  =  (Qi,  E i,6i,  qo,i,  Ff),  where: 

•  Qi  =  V 

•  Ej  =  Si  U  Ci  U  {e}  where  Si  C  S  and  Ci  C  C 

•  Qo ,i  €  V  is  the  unique  CFG  entry  state 

•  Fi  =  {v  £  V  \  qo,i  \v  is  a  CFG  exit  } 

•  q  £  5i(p,  a)  if  a  <  p  and  (p,  q)  £  E 

•  q  £  §i(p,  e)  if  da  £  Si  U  Ci  :  a  f]  p  and  (p,  q)  £  E 

This  definition  simply  labels  CFG  edges  as  described  in 
Section  5.1.  All  local  models  are  e-reduced  and  minimized 
to  reduce  their  storage  requirements. 


The  definition  of  the  global  Dyck  model  depends  upon 
a  classification  of  function  call  sites.  Let  C i,  C2,  C3,  and  C4 
partition  C  as  follows: 

•  a  £  Ci  if  a  does  not  recurse  and  9(a)  must  generate  at 
least  1  system  call  before  returning. 

•  a  £  C2  if  a  does  not  recurse  and  8(a)  may  condition¬ 
ally  generate  a  system  call  before  returning. 

•  a  £  C3  if  a  does  not  recurse  and  0(a)  will  never  gener¬ 
ate  a  system  call  before  returning. 

•  a  £  C4  if  a  may  recurse. 

We  write  C12  to  denote  C\  UC2. 

Definition  3  [Dyck  Program  Model ] 

Let  i  range  over  all  functions  in  Pr  with  r  the  entry 
point  function.  Let  f  and  g  be  the  symbols  used  in  the 
proof  of  Theorem  2,  with  F  =  {/e,7  :  7  £  C 12}  and 
Q  =  {<7<r,T  :  7  £  C 12}.  Then  D  is  a  Dyck  model  if  there 
exists  De  =  (Q,  S  U  {e},  T,  S€,  qo,  Zq,  F)  with: 

1.  Q  =  \jQi 

i 

2.  r  =  C12 

3.  z=su Fug 

4-  qo  =  qo,r 

5.  z0  =  e 

6.  F  =  Ft 

7.  (q,  z)  £  Se(p,  a,  z)  if  a  £  S  and  3  i  :  q  £  Sfp,  a) 

8.  (q,  z)  £  Se(p,  e,  z)  if 

(a)  3a  £  C2  U  C3  3i  :  q  £  Si(p,  a)  ;  or 

(b)  3a  £  C4  3 i  3 r  £  :  r  £  Si(p ,  a)  A  q  =  q0,g(a)  / 

or 

(c)  3a  £  C4  3  i  3  r  £  Qz  :  q  £  Sfr,  a)  A  p  £  F0(a) 

9.  (q,  za)  £  Se(p,  /e,a,  z)  if  a  £  C12  A  3*  3r  £  Qi  :  r  £ 
$i(p ,  a)  A  q  =  g0,e(a) 

70.  (g,  e)  £  Se(p,ge<a,a)  if  a  £  C12  A  3*  3r  £  Q,  :  q  £ 
(5,:(r,a)  Ap  £  Fe(o) 

and  D  =  ( Q ,  S,  T,  5,  qo,  Zo,  F)  is  De  under  e-reduction. 

Several  properties  of  the  definition  require  explanation. 
Property  3  adds  push  and  pop  symbols  to  the  alphabet  of 
system  calls.  Property  7  maintains  the  system  call  transition 
property  of  the  local  automata:  a  system  call  will  not  mod¬ 
ify  stack  state.  Property  8(a)  adds  e-edges  around  call  sites 
that  may  not  generate  a  system  call.  Properties  8(b)  and  (c) 
link  automata  at  recursive  call  sites  with  e-edges  rather  than 
with  edges  that  update  the  PDA  stack.  Properties  9  and  10 
describe  transitions  for  precalls  and  postcalls  that  modify 
stack  state. 


C.2.  Stack-determinism  of  Dyck  Model 

Theorem  4  The  Dyck  model  is  an  sDPDA. 

Proof:  Clearly,  the  Dyck  model  is  a  PDA. 

e  S  =>  Mz  £  T  /Bp  £  Q  :  S(p,e,  z)  0,  so  sDPDA 
condition  1  is  satisfied. 

Suppose  Bqi,q2  £  Q  so  that  for  some  cr  £  S  and  z  £ 
T,  5(qi,  a,  z)  =  (pi,  W\)  and  S(q2,a,z)  =  ( p2,w2 )  with 
w\  w2.  Proof  by  contradiction  in  three  cases: 

1.  If  c r  £  S,  then  w\  =  z  =  w2  by  Property  7. 

2.  If  a  £  F,  then  37  £  C12  :  er  =  /e  7  and  Wi  =  27  = 
w2  by  Property  9. 

3.  If  <7  £  Q,  then  37  £  C12  :  er  =  pe  7.  Then  z  =  z' 7  and 
W\  =  z'  =  w2  by  Property  10. 

Thus,  sDPDA  condition  2  holds.  □ 


